&#34;Black-box&#34; video or still recorder for commercial and consumer vehicles

ABSTRACT

A method and system for monitoring an automobile is disclosed. A centralized black box receives and stores images from different image sensors positioned on the vehicle. In one example, digital pixel sensors are used to provide image data. One of the features in the digital pixel sensors is the multi-capturing image capability that ensures that the image data is non-saturated. As a result, details can be captured in the image data regardless an automobile drives constantly from a bright condition to a dark condition or vice versa.

CROSS-REFERENCE TO RELATED APPLICATION

[0001] This application is related to U.S. app. Ser. Nos. 09/567,786 and09/567,638, both filed May 9, 2000, each of which is hereby incorporatedby reference.

BACKGROUND OF THE INVENTION

[0002] 1. Technical Field

[0003] The invention relates to image sensor systems. More particularly,the invention relates to the application of an image sensor architectureand associated method to an automobile monitoring system.

[0004] 2. Description of the Prior Art

[0005] Traffic accidents are a frequent occurrence. For legal andinsurance reasons, a motorist must determine the cause of the accident.Many times, however, the motorist is confronted by conflictingtestimony, or no testimony at all, regarding the accident. A method andsystem for recording the environment of the car as well as the eventsleading up to the accident would be of great assistance in puzzling outthe causes of an automobile accident after the fact.

[0006] Filming the car's environment while the car is in motion usingstandard videotape machinery is not an attractive option, as thevideotapes storing the information would require a lot of room forstorage. While digital cameras would allow for some decrease in thestorage area required, the digital cameras available nowadays offer lessin the areas of dynamic range and processing time. Both of these areaswould be crucial for such a device to provide the information needed, asthe dynamic range can change quickly and crucial information about anaccident can occur in less than a second.

[0007] What are needed are a system and method for monitoring thesurrounding environment of an automobile that can the store the dataefficiently and react quickly to changing environmental conditions.

SUMMARY OF THE INVENTION

[0008] In one embodiment of the present invention, a method and systemfor monitoring an automobile are disclosed. A centralized black boxreceives and stores information, including images, from differentsensors positioned on the vehicle. These sensors may include, but not belimited to, speed sensors and temperature sensors monitor differentconditions of the automobile, such as speed and engine temperature.According to the present invention, some of these sensors also includedigital image sensors for increased contrast dynamic range and that haveanalog to digital conversion circuits at a pixel level to facilitatemultisampling image capturing.

BRIEF DESCRIPTION OF THE DRAWINGS

[0009]FIG. 1 is a block diagram that shows the invention implementedupon a truck;

[0010]FIG. 2 is a flowchart that shows the method used to store theinformation gathered into memory; and

[0011]FIG. 3 shows an example of a multisampling of an image or multipleexposures of a scene.

DETAILED DESCRIPTION OF THE INVENTION

[0012] In the following detailed description of the invention, somespecific details are set forth to provide a thorough understanding ofthe presently preferred embodiment of the invention. However, it shouldbe apparent to those skilled in the art that the invention may bepracticed in embodiments that do not use the specific details set forthherein. Well known methods, procedures, components, and circuitry havenot been described in detail to avoid unnecessarily obscuring theinvention.

[0013] The present invention uses advances in digital sensory equipmentand memory storage methods and applies them to developing a system andmethod for monitoring land vehicles. The device is not limited toautomobiles and could in fact be applied to any land vehicle, such asmotorcycles, tractors, and construction equipment. In particular, thenew system attaches digital image sensors in strategic locations on aland vehicle. These sensors monitor respectively the environmentsurrounding the vehicle and store generated data into a centralizedstorage unit. In this data storage unit, all data that is within apreviously specified time frame is stored until the happening of anevent, such as a collision with another vehicle. The further intake ofdata is frozen until the data presently stored is reviewed. In oneembodiment, security measures are placed to prevent tampering with thedata.

[0014]FIG. 1 displays one exemplary embodiment for this invention. Acentralized “black box” 110 containing electronic signal processingcircuitry 120 and electronic storage medium 130 is mounted on a vehicle140. The black box system is activated when the vehicle is started andcontinues to run until the vehicle is turned off. During operation, theblack box takes in and stores electronic information from variouselectronic “sensory inputs” or sensors 150. The sensors can be attachedto the automobile in a variety of places beyond the one shown in FIG. 1.

[0015] In one exemplary embodiment, these sensors can includeinformation about the speed the vehicle is traveling, or measurements ofthe state of the engine such as fluid levels or temperature. Many ofthese sensors are readily available in the current technology.Additionally, recent advances in digital pixel sensor (DPS) technologyincrease the viability of including video/still images takenoutside/inside the vehicle by various mounted cameras.

[0016] In one embodiment, the electronic storage medium used in theblack box would be non-volatile solid-state storage, such as flashmemory. Other possible storage mediums include tape-based mediums suchas DV, or disc based storage devices such as MiniDisc, CD or DVD. Tosave on cost, the storage medium would most likely utilize a fixedmemory size, first in first out data storage architecture. Thecontrolling circuit for the black box would direct the data storagemedium to record information until the storage capacity of the medium iscompletely filled.

[0017] Any new information obtained at this point would be recorded overthe least recent information. This method allows for the most recentevents to always be present in the system memory. The storage capacitycould be adjusted to suit the need of the application. For example, amemory buffer could be kept to maintain up to ten minutes of memory. Thebuffer maintains data to reflect the scene ten minutes before anaccident happens.

[0018] In one embodiment, the “black box” system collects theinformation following a method illustrated by flowchart 200 in FIG. 2.The flowchart is presented according to one embodiment of the presentinvention. The order therein shall not be construed as limitations tothe present invention. Different embodiments may result in differentorders of the process in flowchart 200. Once the vehicle has started202, the sensors begin to record the scene surrounding the vehicle 210.The data in this recording is compressed 220. Before the data istransferred to a buffer of a certain size, the process first checks ifthe buffer can accommodate more data 230. If it is determined that thebuffer is full, the data therein is discarded on a first-in-first-outbasis 240. In one embodiment, a fixed amount of data is discarded. Inanother embodiment, an amount of data is discarded to just leave asufficient space to accommodate new data from 220. In any case, thecompressed data from 220 is transferred to the buffer 250. If noaccident occurs 260, the process goes back to 210 to continue monitoringthe surroundings of the vehicle. If an accident does occur 260, therecording process immediately stops 270 or stops after a predeterminedtime (e.g. 2 seconds) provided that the sensing equipment stilloperates.

[0019] Video images or digital still photos of the surroundingembodiment could be stored into the data storage unit.

[0020] Camera modules deployed on the vehicle could use varioussolid-state circuit technologies to collect, process and transmit theimage information back to the base unit. To collect image information,digital pixel sensors (DPS) technology is deployed to accommodatevarious lighting conditions the vehicle may enter. Generally, a vehiclemay drive under very bright or dark conditions. Traditional CCD or CMOSimage sensors would produce saturated images under such dynamicallychanged lighting conditions. One of the features in the DPS technologyis the ability to prevent the digital pixel sensors from producingsaturated images. With its pixel-level analog-to-digital conventioncircuitry, a multisampling process during an image capture is madepossible.

[0021] As disclosed in U.S. App. Nos.: 09/567,786 and 09/567,638, one ofthe features in a digital pixel sensor is the high dynamic rangeachievable via multisampling. A large dynamic range greatly increasesthe useful information in images captured by the camera using thedigital pixel sensor. A low-cost, commercially available digital pixelsensor can have a dynamic range of greater than 100 dB with 16-bitquantization, or 131 dB with 20-bit quantization.

[0022] The DPS for the automobile “black box” may be implementedaccording to U.S. app. Ser. Nos. 09/567,786 and 09/567,638. The DPSdiffers from standard digital picture technology in that each pixelsensor is comprised of not only the photosensitive element, but alsoelement-level analog-to-digital conversion circuits. The element-levelanalog-to-digital conversion circuits alleviate the main cause of longdata readout time on standard digital cameras, the bottleneck createdbetween the photosensitive elements and subsequent data memory. With theimproved readout time in the DPS, multiple image data can be readout tothe data memory during an image capture. To prevent image datasaturation, innovative measures are taken to ensure that only willnon-saturated data be read into the memory.

[0023] In one embodiment, the DPS for the automobile “black box”comprises a photosensitive chip. The photosensitive chip, which istypically fabricated on a substrate such as CMOS, comprises a pluralityof photodetectors that are arranged in a 2-dimensional array. The numberof the photodetectors in the photosensitive chip typically determinesthe resolution of the resulting digital images, with horizontalresolution related to photodetectors per row, and vertical resolutionrelated to photodetectors per column.

[0024] The dynamic range problems occur in most digital cameras becauseone of the photosensitive elements may become saturated with photonswhile another might not receive enough photons to even leave an image.This contrast is especially important in capturing images from a movingautomobile, which might be going constantly from dark to light or viceversa. FIG. 3 shows an example of the multi-sampling image capturing. Itis assumed that an image sensor in a camera mounted on a vehicle is setto expose to a scene for a duration of S (e.g. {fraction (1/60)} second)and 4 samples are to be read. Each of the samples is read out at T, 2T,4T and 8T, wherein 8T=S or T=S/T. As shown in FIG. 3, frame 1 is createdafter time T, frame 2 is created after time 2T accumulating from T,frame 3 is created after time 4T accumulating from T and 2T, and frame 4is created after time 8T accumulating from T, 2T and 4T. One of theadvantages of having multiple images of the same target within anexposure time is the ability to expand the dynamic range of the imagecaptured as pixel values in each of the frames can be selectively readinto the data memory in accordance with a threshold value. Because ofthe relative short exposure time, frame 1 typically captures informationthat is related to high illumination areas in the target. Likewise,because of the relatively long exposure time, frame 4 typically capturesinformation that is related to low illumination areas in the target.Frame 2 and frame 3 thus capture information that is related togradually increased illumination areas in the target. As a result, thecombination of the multiple images provides a very wide dynamic range.

[0025] The above multisampling imaging, i.e. multiple images of the sametarget within an exposure may be appreciated from another perspective.When the target has an extremely high contrast, namely, the bright areais very bright and the dark area is very dark, a single exposure settingcould hardly generate an image including details both in the bright anddark areas. A more ideal solution would be to create a single image fromtwo exposure settings, one for the bright area and one for the darkarea, which is practically difficult to achieve with the currenttechnology. With the present invention, the two or more exposuresettings are “simulated” by accumulating exposure time of the sametarget and generating multiple images thereof. A composition of themultiple images with respect to existing values of each pixel leads to afinal image including details both in the bright and dark areas.

[0026] In operation, when a value in the threshold memory cell detectsthat a corresponding photodetector is saturated, the accumulated chargefrom the corresponding photodetector in subsequent frames will beprevented from being written into the corresponding data memory cell.This feature is very important to the imaging for the automobile“blackbox” and is believed a significant departure from a conventionalimage sensor.

[0027] At the end of an exposure, image data in the data memory istransmitted to a memory device in the “black box”. To improve imagequality or reduce the amount of image data being sent back to the memorydevice, image processing and compression may be done in the cameramodules. To transmit the data back to the black box, various techniquesmay be deployed such as direct wire connection, or utilization of awireless protocol such as Bluetooth.

[0028] The advantages of the invention are numerous. Differentembodiments or implementations as a method, a system and an apparatusmay yield at least one or more of the following advantages. Oneadvantage of the invention is that the present invention now makes itfeasible for an automobile to be equipped with imaging sensors thatrecord surrounding conditions of the automobile prior to an accident.Another advantage of the invention is the employment of one or moredigital pixel sensors that can produce non-saturated image dataregardless of the lighting conditions in which a vehicle operates. Manyother features and advantages of the present invention are apparent fromthe written description, and thus, it is intended by the appended claimsto cover all such features and advantages of the invention. Further,since numerous modifications and changes will readily occur to thoseskilled in the art, it is not desired to limit the invention to theexact construction and operation as illustrated and described. Hence,all suitable modifications and equivalents may be considered to fallwithin the scope of the invention.

1. A monitoring system for a ground transportation vehicle, the systemcomprising: an imaging device employing a digital pixel sensor formonitoring an environment of the vehicle, the digital pixel sensorproducing multiple images within a predetermined exposure time, theimages read out respectively at a set of time periods within theexposure time to produce digital image data, wherein a longest timeperiod of the set of time periods does not exceed the exposure time; acontrolling circuit for initiating and terminating a measurementactivity for the imaging device; and an electronic storage medium forstoring a succession of the digital image data from the digital pixelsensor.
 2. The monitoring system of claim 1, wherein the digital pixelsensor comprises an array of photosensitive elements, each of thephotosensitive elements coupled to an analog to digital conversioncircuit.
 3. The monitoring system of claim 2, wherein each of thephotosensitive elements couples to a memory cell so that data generatedby the each of the photosensitive elements can be directly written intothe memory cell.
 4. The monitoring system of claim 1, wherein thedigital pixel sensor includes data memory that is coupled to theelectronic storage medium.
 5. The monitoring system of claim 4, whereinthe electronic storage medium is configured to store image data for apredefined period, and wherein the electronic storage medium isconfigured to dump least recent image data when receiving latest imagedata.
 6. The monitoring system of claim 5, wherein the electronicstorage medium is configured to stop receiving image data after acertain time if the monitoring system still operates.
 7. The monitoringsystem of claim 1, wherein the controlling circuit initiates ameasurement by the imaging device at a regularly spaced interval.
 8. Themonitoring system of claim 1, wherein the controlling circuit terminatesmeasurements by the imaging device in the event of damage to the groundtransportation vehicle.
 9. The monitoring system of claim 1, wherein thecontrolling circuit terminates recording by the electronic storagemedium in the event of damage to the ground transportation vehicle. 10.The monitoring system of claim 1, wherein the electronic storage mediumutilizes a first in first out data storage architecture.
 11. Themonitoring system of claim 10, wherein the electronic storage medium isone of a non-volatile solid-state storage device, a tape based medium,and disc based storage device.
 12. The monitoring system of claim 1,wherein the electronic storage medium has a security device implanted toprevent tampering image data therein
 13. A ground transportation vehiclemonitoring method, the method comprising: collecting image informationabout surrounding driving conditions of the vehicle, wherein the imageinformation is produced in an imaging device employing a sensor capableof generating multiple images within a predetermined exposure time, theimages read out respectively at a set of time periods within theexposure time to produce the image information, and wherein, a longesttime period of the set of time periods does not exceed the exposuretime; transmitting said image information to a base unit on the vehicle;and storing said information in an electronic storage medium.
 14. Themethod of claim 13, wherein the vehicle is equipped with a plurality ofsuch imaging device, each placed on a designated part of the vehicle.15. The method of claim 14, wherein the image information is compressedin accordance with a compression scheme.
 16. The method of claim 13,wherein the electronic storage medium is configured to have a capacityfor a predetermined period.
 17. The method of claim 16, wherein thestoring of the image information comprising discarding least recentstored image information when receiving the image information from theimaging device.
 18. The method of claim 17, wherein the storing of theimage information is immediately stopped in the event of damage to theground transportation vehicle.